Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs.

Weaknesses: [Empirical results are not particularly strong] In Table 1, second block SemiNAS (RE) barely outperforms RE. Also in Table 1, third block, SemiNAS outperforms NAO only by 0.12%. While these numbers come from NASBench-101, we all know that empirically, CIFAR-10 accuracy of *any* models and experiments have a standard deviation of about 0.05 - 0.1, and hence I would say there is no improvement between SemiNAS and other baselines. Meanwhile, Table 2 also includes some irrelevant baselines, namely those in the second block, while omits an important baseline. EfficientNet-B0 (Tan and Le, 2019) was found in the same search space of MobileNet-V2, has 390M FLOPs, and has the ImageNet top-1 error of 23.2, which is about the same as 23.5 of SemiNAS. Why is this not reported?

The paper proposes an interesting semisupervised approach to neural architecture search: Using architecture accuracy prediction function to to train the controller (architecture generator), and shows that such approach yields efficiency improvements. Reviewers generally agree on simplicity of this method and good experimental evaluation. Reviewers 3, 4 point out a number of missing comparisons however many of these are addressed in the rebuttal. It would also be good to understand why this method work, since as reviewer points out, no new information is added by the evaluation network - which on the other hand makes the experimental confirmation interesting. Overall this is an interesting and simple method with good evaluation and results.

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them.

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose \emph{SemiNAS}, a semi-supervised NAS approach that leverages numerous unlabeled architectures~(without evaluation and thus nearly no cost) to improve the controller. Specifically, SemiNAS 1) trains an initial controller with a small set of architecture-accuracy data pairs; 2) uses the trained controller to predict the accuracy of large amount of architectures~(without evaluation); and 3) adds the generated data pairs to the original data to further improve the controller. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. 2) It achieves higher accuracy under the same computational cost. On NASBench-101 benchmark dataset, it discovers a top 0.01% architecture after evaluating roughly 300 architectures, with only 1/7 computational cost compared with regularized evolution and gradient-based methods. On ImageNet, it achieves 24.2% top-1 error rate (under the mobile setting) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively. Our code is available at https://github.com/renqianluo/SemiNAS.